Aiming at the problems of blurred target boundaries and low image segmentation efficiency in the image segmentation process of ancient murals, a multi-class image segmentation model fused with a lightweight convolutional neural network named MC-DM (Multi-Class DeepLabV3+MobileNetV2 (Mobile Networks Vision 2)) was proposed. In the model, DeepLabV3+ architecture and MobileNetV2 network were combined together, and the unique spatial pyramid structure of DeepLabV3+ was utilized to perform multi-scale fusion of the convolutional features of the mural to reduce the loss of image details during the mural segmentation. First of all, the features of the input image were extracted by MobileNetV2 to ensure the accurate extraction of image information and reduce the time consumption at the same time. Secondly, the image features were processed through the dilated convolution, so that the receptive field was expanded, and more semantic information was obtained without changing the number of parameters. Finally, the bilinear interpolation method was utilized to up-sample the output feature image to obtain a pixel-level prediction segmentation map, so that the accuracy of image segmentation was ensured to the greatest extent. In the JetBrains PyCharm Community Edition 2019 environment, a dataset made of 1 000 mural scanning pictures was used for testing. Experimental results showed that the MC-DM model had a 1% improvement in training accuracy compared with the traditional SegNet (Segment Network)-based image segmentation model, and had a 2% improvement in accuracy compared with the image segmentation model based on PSPNet (Pyramid Scene Parsing Network), and the Peak Signal-to-Noise Ratio (PSNR) of the MC-DM model was 3 to 8 dB higher than those of the experimental comparison models on average, which verified the effectiveness of the model in the field of mural segmentation. The proposed model provides a new idea for the segmentation of ancient mural images.